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Intrinsic Properties of Complete Test Snites 


Adilson Luiz Bonifacio* Arnaldo Vieira Moura^' 


Abstract 

Completeness is a desirable property of test suites. Roughly, completeness guarantees that a non¬ 
equivalent implementation under test will always be identified. Several approaches proposed sufficient, 
and sometimes also necessary, conditions on the specification model and on the test suite in order to 
guarantee completeness. Usually, these approaches impose several restrictions on the specification and 
on the impiementations, such as requiring them to be reduced or compiete. Further, test cases are 
required to be non-biocking — that is, they must run to compietion — on both the specification and the 
impiementation modeis. In this work we deaf test cases that can be biocking, we define a new notion that 
captures completeness, and we characterize test suite completeness in this new scenario. We establish an 
upper bound on the number of states of implementations beyond which no test suite can be complete, 
both in the classical sense and in the new scenario with blocking test cases. 


1 Introduction 

Completeness of test suites has been largely studied for models based on Finite State Machines 
(FSMs) |BMdSS121 lHU02l lUFFYObl [HFTOl IBJVlMal I1JWZ971 ISFY12| . A test suite is called complete for 
a FSM specification when it provides complete fault coverage |BMdSSi^ IHU02) . Several works have pro¬ 
posed strategies for generating complete test suites |dSSPY09] , or for checking if a given test suite is complete 
for a given specification [BM14a) . Some of them presented necessary conditions [PB961 IYPvB94] for test 
suite completeness, whereas other approaches gave sufficient, but not necessary, conditions for test suite 
completeness [DEFYOSl IPYOOi ISPlOi IUWZ97) . Some more recent works have described necessary and suf¬ 
ficient conditions for test suite completeness |BM14al ldSSPY09] . All these works imposed restrictions on 
the specification and implementations, or over the fault domains [DEFYOSl IPYOOI ISPIOIIUWZ971 IBM14a] . 
Some of them considered specifications with n states and restricted the implementations under test to have 
at most n states. Further, in some approaches specification and implementations are required to be reduced 
or completely specified machines. Always, test cases have been required to be non-blocking on both the 
specifications and the implementations models. This meaning that all test cases are assumed to run to 
completion in these models. In particular, even if implementations are treated as black boxes, all test cases 
are assumed to run to completion on implementations. 

In this work we deal with the more general scenario where test cases can be blocking. In particular, we do 
not require that all test cases run to completion when implementations can be partial FSMs, and are treated 
as true black boxes. We propose a new notion of equivalence, called “alikeness”, and we extend the classical 
notion of equivalence when blocking test cases can be present, thus giving rise to the notion of “perfectness”, 
in lieu of the classical notion of completeness. We then use bi-simulation relations and reducibility over 
machines to characterize test suite perfectness in this new more general scenario. 

A related issue that concerns test suite completeness is the maximum size of implementations that can 
be put under test. Usually, earlier works constrained implementations to have at most the same number of 
states as the given specification. We are not aware for any work that gives a precise relationship between the 
maximum number of states in implementations and the size of test suites in order to get positive verdicts 
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when such implementations are put under test. Here, we establish a precise upper bound on the number of 
states of implementations under test, beyond which no test suite can be complete, both in the classical sense 
and in the more general scenario when blocking test cases can be present. The bound is based on test suite 
size and the number of states in the given specification. 

We organize the paper as follows. Basic definitions and notations appear in Section [H Section [3] gives 
the perfectness of test suites in terms of the property of isomorphism between machines. We relate the 
well-known notion of completeness to the notion of perfectness in Section |4l In Section [5] we establish an 
upper bound on the number of states in candidate implementations beyond which no test suite is complete. 
Section[S]defines the notion of m-perfectness, where m is the number of candidate implementations. Section[7] 
states some conclusions. 


2 Definitions and notation 

Let X be an alphabet. The length of any finite sequence a of symbols over X is indicated by |a|. The empty 
sequence will be indicated by e, with jej = 0. The set of all sequences of length k over X is denoted by 
X^, while X* names the set of all finite sequences over X. When we write a = XiX 2 ■ ■ ■ Xn & X* (n > 0) we 
mean Xi € X {1 < i < n), unless noted otherwise, and similarly for other alphabets. Given any two sets of 
sequences A,BC X*, their symmetric difference will be indicated by AqB, that is AqB = (Ani?)U(HnH), 
where A indicates the complement of A with respect to X*. The usual set difference is indicated by A\B. 

Remark 1 Aq B = $ ij^ A = B. 


2.1 Finite state machines and test snites 


Next, we write the definition of a Finite State Machine |BM14al IGil62) . 
Definition 1 A FSM is a system M = (S, so,X,0, D, S, X) where 

• S is a finite set of states 

• So € S is the initial state 

• X is a finite set of input actions or input events 

• O is a finite set of output actions or output events 

• D Q S X X is a specification domain 

• S : D ^ S is the transition function 


• X : D ^ O is the output function. 


In what follows M and N will always denote the FSMs {S,so,X,0,D,S,X) and {Q,qo,X,0', D', yL,T), 
respectively. Let a = xiX 2 ■ ■ ■ Xn G X*, uj = 0102 • • • a„ € O* (n > 0). If there are states rt G S {0 < i < n) 

such that S(ri_x,Xi) = and X{ri_i,Xi) = Ui (1 < i < n), then we may write xq r„. When the input 

• • 1 • ^1 

sequence cr, or the output sequence w, is not important, then we may write tq ^ r^, or rg —)• r„, respectively, 
and when both sequences are not important we may write rg —>■ We can also drop the target state, and 

write rg or tq —>■ . It will be useful to extend the functions 5 and A to pairs (s,cr) G S x X* . Let 


D = 




G X* 


s G 


q}- 


Define the extensions 5 : D ^ S and A : D —5> O* by letting S(s,a) = r 


and A(s,ct) = uj whenever s -U r. When there is no reason for confusion, we may write D, S and A instead 
of D, S and A, respectively. Also, the function U : S ^ X* will be useful, where U{s) = {cr | (s, cr) G D}. 
Informally, U{s) denotes all input action sequences that can be run from the state s. 

Now we are in a position to define test cases and test suites. 


^ Here, ‘iff’ is short for ‘if and only if’. 


2 






Definition 2 Let M be a FSM. A test suite for M is any finite nonempty subset of I*. Any element of a 
test suite is a test case. 

Before we can define test completeness, we need the classical notions of distinguishability and equivalence. 

Definition 3 Let M and N be FSMs and let s € S, q G Q. Let C C X*. We say that s and q are C- 
distinguishable iff \{s,a) fi- T{q,a) for some a G U{s)^\U{q)f^ C, denoted s 9- Otherwise, s and q 
are C-equivalent, denoted s q. We say that M and N are C-distinguishable iff sq do, o.nd they are 
C-equivalent iff sq ssc go- 

When C is not important, or when it is clear from the context, we might drop the index. When there is 
no mention to C, we understand that we are taking C = X*. In this case, the condition U{sq) H 17(go) n C 
reduces to U{so) r\U{qo). For the ease of notation, we also write M Kic ^ when M and N are C-equivalent, 
and M N when they are C-distinguishable. 

Now we can state the conventional notion of a m-complete test suite. 

Definition 4 Let M be a FSM and T a test suite for M. Let m > 1. Then T is m-complete for M iff for 
any FSM N, with U{so) C U{qo) and with at most m states, if M ffi N then M 567 - N. 

Note that if a runs to completion from sq, that is, sq —i, then a must also run to completion from go, that 

is we must have go —i. The definition says that any discrepancy between the behaviors of the specihcation 
M and any implementation N will be detected if we run the tests in T through M and TV, provided that 
we consider implementations with at most m states. Note that the technical condition U{so) C C/(go) will 
always be satisfied if we were to test implementations that were complete FSM models. A FSM M is said 

x/ 

to be complete when D = S x I, that is, for any state s and any input symbol x, we always have s —^ . 

2.2 The notion of ‘alikeness’ 

A blocking test case for M is a sequence a 0 U{so), otherwise we say that cr runs to completion in M. Then, 
given two FSM models M and A^, if cr G C(so) 0 U{qo), either a blocks in M and runs to completion in 
N, or vice-versa. Given a test suite T and two FSM models M and N ,we want to say when M and N are 
equivalent in some more general sense, that is, even considering that we may have blocking test cases, for 
M or N, in T. Intuitively, all cr G X that is a blocking test case for M must also be a blocking test case for 
N, and vice-versa. Furthermore, any test case that is non-blocking for both M and N must output identical 
behaviors when run through both models. In this case M and N will be said to be T -alike. 

Definition 5 Let M and N be FSMs and let s G S, q G Q. Let CGI*. We say that s and q are C-alike, 
denoted s ~c q, iff {U{s) 0 C/(g)) fl C = 0 and A(s, cr) = T{q,a) for all a G C/(s) fl 17(g) n C. Otherwise, 
s and q are C-unlike, denoted s 'fic g- We say that M and N are C-alike iff sq qo, otherwise they are 
C-unlike. 

We may also write M when M and N are C-alike, or M N when they are C-unlike. Again, 

when C is not important, or when it is clear from the context, we might drop the index, and when there is 
no mention to C, we understand that we are taking C = X*. 

Remark 2 We note of the following simple observations. 

1. Using RemarkUl we note that s ^ q is equivalent to U{s) = U{q) and A(s,ct) = T{q,a) for all a GU{s). 

2. If Cl C C 2 , then s ^C 2 9 implies s 9 - 

3. If s ^ q, then s '^c d, for all C Cl*. □ 
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An important aspect of the alikeness relation, is that it is an equivalence relation when M and N 
are the same machine, that is, when is defined over a single set. We note that this is not the case, in 
general, with the distinguishability relation rsc- 

Lemma 1 Let M be an FSM and let C Q I* . Then '^c is an equivalence relation on S. 

Proof Let s,r,p G S he states of M. We clearly have U{s) Q U{s) = 0 and A(s, a) = A(s, a) for all 
a G U{s) Cl C. So, is reflexive. Also, set intersection, the symmetric set difference © and, of course, 
equality are commutative. Hence, '^c is symmetric. 

For transitivity, assume s r and r '^c P- Let a G U(s)r]C. Thus a G U{r) because s r, and then 
a G U{p) because r p. So, U(s) C [7 (p). Since we already have symmetry, we get p '^c ^ and r s, and 
a similar argument gives U(p) C C/(s), showing that (U{s) 0 U(p)) fl C = 0. Now, let a G U {s) r\U (p) f] C. 
Since s we get a G U{r) and so A(s,a) = A(r, a). But also r p, and so A(r, a) = A(p, a), thus 

establishing A(s, ot) = A(p, ot). We may then conclude that s '^c P, and is transitive. □ 

Remark 3 We note that, in Lemma\l\ the argument establishing the transitivity of the alikness relation 
is still valid when it is defined as a relation between the states of two distinct machines. 

When reducing FSMs in the presence of blocking test cases, we will need the following technical result. 

Lemma 2 Let M be a FSM and let s,r G S be states of S, with s ^ r. 

, . X I a /7~\ 1 x/a . ^ 

(1) If s ^ p With X and a € O, then r ^ q with p ^ q, for some q € o. 

. \ a/uj a/uj . ^ 

(2) If s ^ p with a € X and lj £ O , then r ^ q, with p ^ q for some q £ o. 

X j b 

Proof We first treat item 1. We have x G U{s), and so x G U{r) because s ~ r, which leads to r —>• g for 
some q G S, b G O. Now, a: e C/(s) n U{r) and, since s ~ r, we get a = A(s,a:) = X{r,x) = b. It remains 
to show that p ^ q. Let a G U(p). Then xa G U{s), and again xa G U(r). Since M is deterministic, 
this gives a G U{q), and so U{p) C U{q). Using Remark [Sfl) we have r ^ s, and a similar argument gives 
Uiq) U U{p). We conclude that U{p) = U{q), and so U{p) 0 U{q) = 0. Now, let fi GU{p) C\ U{q). Then, 
xf) G U{s) Cl U{r), and since s ^ r this gives aX{p,a) = X{s,xfd) = X{r,xf3) = aX{q,a). We conclude that 
A(p, a) = X{q,a), as desired. 

Now, item (2) follows by a simple indiction on |a| > 0, and using the result of item 1. □ 

The notion of perfectness has been introduced by Bonifacio and Moura [BM14bl IBM13) , in order to cope 
with test cases that may not run to completion either in the specification or in the implementation models. 
It is based on the notion of alikness. 

Definition 6 I [BM14b| i Let M be a FSM and T be a test suite for M. Then T is perfect for M iff for 
any FSM N, if M N then M ybj. N. 

That is, when T is a perfect test suite for a specification M, then for any implementation under test N, if 
M and N are unlike, then they are also T-unlike. 

In Definition | 6 l there is no limit in the size of the implementations. In the next definition, the key 
property of M N implying M N is required to hold only for implementations with up to a predefined 
number of states. 

Definition 7 Let M be a FSM, let T be a test suite for M, and let m > 1. Then T is m-perfect for M iff 
for any FSM N with at most m states, if M N then M N. 
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2.3 Simulations and perfectness 

In [BM14bl IBM13) bi-simulation was used to characterize test suite perfectness. 


Definition 8 Let M and N he FSMs. We say that a relation R C S x Q is a simulation (of M hy N) iff 

"3^ I a 31 j a 

(so, go) G ti, and whenever we have (s, q) G R and s ^ r in M, then there is a state p G Q such that q ^ p 
in N and with {r,p) G R. We say that M and N are bi-similar iff there are simulation relations Ri C S xQ 
and R 2 Q Q X S . 


The following simple facts will be used later. 

Fact 1 The simulation relation is transitive, that is, let Mi = (Si, Si,T,0, Di, 6 i, Xt) be FSMs, i = 1,2,3, 
and where M 2 simulates Mi and M 3 simulates M 2 . Then, M 3 simulates Mi. 

Proof Let Ri C Si x S 2 and R 2 Q S 2 x S 3 be simulation relations. Define R C Si x S 3 hy {s,p) S i? iff 
(s, q) G Ri and {q,p) G i? 2 , for some q G S 2 . Firstly, since (si, S 2 ) G Ri and (s 2 , S 3 ) G R 2 we get (si, S 3 ) G R, 

as needed. Moreover, let (s,p) € R and s ^ si. We must have (s,g) G i?i and {q,p) G R 2 for some q G S 2 . 

311 a 311 a 

Since Ri is a simulation, we get q —>■ qi, with (si,gi) G Ri. Since R 2 is a simulation, we get p ^ pi with 
(gijPi) G i? 2 . Then, (si,pi) G R, as desired. 

Fact 2 Let M and N be FSMs, and let R G S x Q be a simulation of M by N. If (s, q) G R and S{s, a) = r 
for some a G I*, then p{q, a) = t with (r, t) G R, for a unique t G Q. 

Proof An easy induction on |q;| > 0. Such a t G Q is unique, since N is deterministic. □ 

Fact 3 Let M and N be FSMs, let R Q S xQ be a simulation of M by N, and let L C Q x S be a simulation 
of N by M. Let {s,q) G R, (q,s) G L, and a G I*. If5{s,a) = r, then p{q,a) = t with {r,t) G R and 
{t, r) G L, for a unique t G Q. 

Proof From d{s,a) = r and (s,g) G R Fact [2] gives a unique t G Q with p{q,a) = t and (r, t) G R. From 
(g, s) G L and p{q,a) = t, Fact [2] again gives some p G S with {t,p) G L and (5(s,a) = p. Since M is 
deterministic and we already have S(s, a) = r we conclude that p = r. Hence, {t, r) G L as desired. □ 


The next lemma shows a useful relationship between bi-simulations and alikeness. 

Lemma 3 Let M and N be FSMs, let R G S x Q be a simulation of M by N, and let L G Q x S be a 
simulation of N by M. Let {si,q) G R and {q,Si) G L, i = 1,2. Then, Si ^ 52 - 


Proof For the sake of contradiction, assume that si 7 ^ S 2 . Definition [5] gives some Vi G S, Oi G O {i = 1,2), 

cx. j 

X GX, and some a GX* with Si ^ Vi {i = 1,2), and such that for some ti,t 2 G S, either 

( 1 ) ri A* ti, i = 1 , 2 , and oi 7 ^ 02 ; or 

(2) ri ti, and x ^ U(r 2 )', or 

(3) r 2 t 2 , and x ^ U(ri). 


From {si,q) G R and Si ri. Fact [2] gives Ui G Q such that g 
N is deterministic, we get ui = U 2 = u and so iri,u) G R (* = 1,2). 

Now, if case (1) holds, then from (rj,u) G R and using Definition [8] we get u 
{i = 1,2). Again, since N is deterministic, we obtain oi = 02 , a contradiction. 

Assume that case (2) holds. Since {ri,u) G R and ri ti. Definition [8] gives u "^A^ Vi, for some 




Ui and (ri,Ui) G R, for * = 1,2. Since 

X j ai 


Vi for some Vi G Q 


Vi G Q. From g 


U 2 and (g, S 2 ) G L, FadjS] gives some G S' with S 2 A and {u 2 ,r 2 ) G L. But we 


Ct j 

already have S 2 ^ ''’ 2 , and so the determinism of M gives r'^ = r 2 . Hence, {u 2 ,r 2 ) G L and then {u,r 2 ) G L 
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because U 2 = u. But we also have u —Vi and so, using Definition |51 we get x G D(r2), contradicting the 
hypothesis of case (2). 

Case (3) also leads to a contradiction, by a reasoning entirely analogous as was done for case (2). 

We conclude that, in fact, si ^ S2, as desired. □ 

The following result establishes a necessary and sufficient condition for perfectness. 

Theorem 1 f [BM14b| l Let M be a FSM and T be a test suite for M. Then T is perfect for M iff any 
T-alike FSM is bi-similar to M. 

In the next section we show that the bi-similarity test can be exchanged for an isomorphism test. 


3 Perfectness and Isomorphism 

In this section we characterize perfectness in terms of isomorphisms between FSMs. 

3.1 Bi-simulation and isomorphism 

Two FSMs are said to be isomorphic when they specify exactly the same model, except for a state relabeling. 

Definition 9 Let M and N he FSMs with 0 = 0'. An isomorphism (of M into N) is a bijection f : S ^ Q 
such that 

1 . /(so) = go; and 

^ j (T ^ j CL 

2. s ^ r in M if and only if f{s) —>■ /(r) in N, for all x £l, a £ O. 

Machines M and N are isomorphic iff there is an isomorphism of M into N. □ 

Remark 4 Let M and N he FSMs. The following are immediate consequences: 

1. f is an isomorphism of M into N if and only if f~^ is an isomorphism of N into M. 

2. Any isomorphism of M into N is also a simulation of M by N. 

The first half of the characterization is easily obtained. 

Lemma 4 Let M and N be isomorphic FSMs. Then, M and N are bi-similar. 

Proof Using Remark [U we have a simulation of M by N, and vice-versa. □ 

Now let M and N be bi-similar. It is clear that if all states in M are unlike, but N has two distinct 
states that are alike, then it is possible for M and N not to be isomorphic, since these two distinct equivalent 
states in N would have to correspond to a single state in M. Machines illustrated in Figures [T] and [Hare a 
case in point. The problem, of course, is that states qi and 52 in have exactly the same blocking input 
sequences and, moreover, the behaviors of qi and (72 in 7Vi are exactly the same under any input sequence a 
that is non-blocking for both of them. 

In the classical sense, a FSM M is reduced if every pair of distinct states in S are distinguishable. When 
treating partial FSM, however, we need also to take into consideration blocking input sequences. In order to 
differentiate from the classical notion of reduction in FSMs, we name reduction in the presence of blocking 
sequences as p-reduction. Both definitions are very similar. 

Definition 10 A FSM M is reduced iff every pair of distinct states of S are distinguishable, and for all 
state s £ S there is a a £l* with S{so,a) = s. □ 
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Figure 2: Specification FSM M. 


Definition 11 A FSM M is p-reduced iff any no two distinct states in M are alike and, moreover, for all 
s € S there is a € I* with (5(so, a) = s. □ 

Hence, for any two distinct states s and r in M there is an input sequence that is a blocking sequence for 
one of them and is not blocking for the other, or there is an input sequence that is non-blocking for both s 
and r but yields different behaviors when starting at the two. Returning to Figures [I] and [5J we see that the 
presence of qi and q 2 in Ni shows that it is not a p-reduced FSM. 

Remark 5 If M is a reduced FSM with at least two reachable states, then there always exists a transition 
out of any reachable state s, that is {s,x) € D for some x Otherwise, s could not be distinguished from 
any other reachable state in M. 

We proceed to show, by a series of simple facts, that if M and N are bi-similar and p-reduced, then they 
are isomorphic. We start by noting that the bi-similarity condition gives two simulation relations R C S x Q 
and L C Q X S. Define a relation f C S x Q as follows: 

(s, q) € f iff So ^ s and qo -i q, for some a £ X*. 

Fact 4 If (s, q) G f then (s, q) G R and {q, s) £ L. 

Proof Observe that (s,g) £ / gives sq ^ s and go ^ q- Since (so, go) S R, Fact [2] gives go ^ p and 
(s,p) £ i?, for some p G Q. Since N is deterministic, we get p = q, and so (s, q) G R. A symmetric argument 
gives (g, s) G L. □ 

Now we show that / is, in fact, a bijection. This will establish that M and N are isomorphic, when they 
are p-reduced. 

f is a function: Let (s,gi) £ /, f = 1,2. From Fact |4] we obtain (s,gi) £ R and (gi,s) G L, i = 1,2. Using 
Lemma [3l we conclude that gi ~ g2. Because N is p-reduced. Definition 1111 forces gi = g2. 

f is total: Let s £ S'. Since M is p-reduced. Definition [TTl gives a £ I* such that So -A s. Since (so,go) G R, 

“/ 

FactOgives go ^ g for some q G Q. Thus, (s,g) £ /. 
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Q;/ 

/ is onto: Let q G Q. Since N is p-reduced, Definition [TT] gives a Gl* such that go ^ q. Since (qo,so) G L, 
“/ 

FactlDgives sq —t s for some s G S. Thus, {s,q) G f. 

f is one-to-one: Let (Si,g) G f, i = 1,2. Using Fact S] we get (si,g) G R and {q,Si) G L, i = 1,2. Then 
Lemma 13] gives si ^ S2- Thus si = S2, since M is p-reduced. 

f is a bijection: We have shown that / is a total function, which is also onto and injective. 

We can now state the main result of this section. 

Theorem 2 Let M and N be p-reduced FSMs. Then, M and N are bi-similar if and only if M and N are 
isomorphic. 

Proof If M and N are isomorphic then they are bi-similar by Lemma [H The argument just given establishes 
the converse. □ 

The next corollary exposes a strong relationship between perfectness of a test suite T for a FSM M and 
p-reduced FSMs that are T-alike to M. 

Corollary 1 Let M be a p-reduced FSM and T be a test suite for M. If T is perfect for M then any 
p-reduced T-alike FSM is isomorphic to M. 

Proof Assume that T is perfect for M and let be a p-reduced FSM that is T-alike M. By Theorem [I] 
we know that N is bi-similar to M. Then, M and N are isomorphic, using Theorem |31 □ 

3.2 p-reduced Finite State Machines 

The converse of Corollary [T] actually also holds. But, since Theorem [2] stipulates that all T-alike FSMs 
must simulate the specification M , first we must show that any FSM can be p-reduced without loosing the 
T-alikness property. 

Recall from Lemma [1] that ^ is an equivalence relation on S' on M. We denote by [s] the equivalence 
class of s under the relation We now use the classical idea of taking quotients in order to construct a 
FSM M that is p-reduced and alike to M. Define 

S = {[s] I s S S, and s some a G I* , uj G O^}, 

and So = [sq]. Next, if s ~ r and {s,x) G D, then Lemma |2]Jl) gives {r,x) G D. We can then define 
D={{ [s],a:) I(s, x) G D}. Since ([s],a:) G D implies {s,x) G D, and Lemma |2jl), again, would give 

5{s,x) ~ S{r,x) for all r G [s], we can define i5([s],x) = [<5(s,x)]. Finally, note that if s r and s p, 

for some p G S, x G I and a G O, then Lemma |3]^1) gives r q, for some q G S, that is, X{s,x) = X{r,x) 
whenever s ^ r and x G U{s). Thus, we can define A([s],a;) = A(s,a::). The construction of M is complete. 

Definition 12 Let M be a FSM. Then M = {S,so,I,0,D,S,X) is the FSM given by the preceding con¬ 
struction. 

The foregoing construction satisfy a number of simple properties that will be useful later. 

Fact 5 Let s,r G S, and let a G I*, w G O*. If s r, then [s] [r]. 

Proof Assume that s r, with x G I and a G O. Then 5{s, x) = r and A(s, x) = a. From the construction 

of M we get 5([s],a:) = [r] and A([s],a;) = a. Hence, [s] [r], and the result follows by an easy induction 

on |q;| >0. □ 


Fact 6 Let r,q € S, and let a G I*, ui G O*. If [r] [g], then ri -U- qi, for some ri,qi G S with r ^ ri 

and q ^ qi- 

Proof Assume that [r] ^ [ 9 ], with x G I and a G O. Then 5([r],a;) = [q] and A([r],a:) = a. From 
(5([r],a;) = [g], the construction of M gives ri,qi G S with S{ri,x) = qi, ri ~ r and <71 ^ q. From 
A([r],a;) = a, we get r2 G S with \{r2,x) = a and r 2 r. Hence, ri ^ r^. 

Since ri -G q\^ this gives -G ra, for some ra G S. But A(r 2 ,x) = a, and so a = b because machines 

are deterministic. Collecting, we have ri -4 gi, ri ~ r and qi ~ q. The result now follows using a simple 
induction on |q;|. □ 

Lemma 5 Let M be a FSM and s,r G S. Let M he the FSM in Definition\lli If [s] [r], then [s] 9 ^ [r]. 

Proof Assume [s] ~ [r] and show that s ^ r. First, we show that U{s) © U{r) = 0. Let a G U{s). Then 
s p, for some p G S and uj G O*. Using Fact [SJ we get [s] [p]. Since [s] ^ [r], Lemma [T] gives 

[r] [g], for some [q] S D. Using Fact [5] we obtain ri gi, for some qi G S with ri r. Hence, Lemma[T] 

now gives r g 2 , for some g 2 G S. We conclude that a G U(r), thus establishing that U{s) Q U{r). A 
similar argument gives U{r) C U{s), and so U{s) = U{r), as needed. To finish, let now a G U{s) r\U{r). 

Then, s p, for some p G S. Repeating the preceding argument would give, again, r -U r 2 , for some 
r 2 G S. Hence, A(s, a) = oj = A(r,uj). From Definition [5] we conclude that s ^ r. □ 

At this point, we can already establish that M is p-reduced. 

Corollary 2 Let M be the FSM in Definition \lll Then, M is p-reduced. 

Proof Let [s] G S. By construction, sq s, for some a G X*, w G O*. Hence, Lemmam2) gives Jq [s], 
because Jq = [sq]- Further, if [s] and [r] are distinct, Lemma [5] implies [s] / [r]. □ 

In the next result, we use the same symbol, to denote the alikeness relations between states of M, and 
also between states of M and of M. The context will always make clear which relation we are referring to. 

Lemma 6 Let M he a FSM and s,r G S. Let M he the FSM in Definitional^ If s r, then s ^ [r]. 

Proof We first show that [/(s) ©17([r]) = 0. Let a G U{s). Since s ~ r, Lemma[2j2) gives a G U{r). Hence, 
using Fact [5] we obtain a G U{[r]), and so U{s) C 17([r]). Conversely, let a G U([r]). Then, Fact |6] gives 
a G C/(ri), where ri ^ r. Thus, ri ^ s, and so using Lemma[2lj2) we get a G U{s). This shows [/([r]) C U{s) 
and we may conclude that U{s) = U{[r]). Hence, 17(s) 0 17([r]) =0 using Remark [U as desired. 

Now, let a G t7(s) n 17([r]). Then, s Si, for some Si G S, uj G O*, and also [r] ^ [ri], for some 
[ri] G S', p G O*. In order to get A(s,a) = A([r],a) we just show that ui = p. From s r, and using 
Lemma[U2), we have r r 2 , for some r 2 G S with r 2 ~ si. Hence, by Fact[S]we get [r] [r 2 ]- The 

determinism of M now gives lu = p. □ 

We can now say that the p-reduction construction preserves alikeness. 

Corollary 3 Let M be a FSM and let M be the FSM in Definition \ll[ Then, M ^ M. 

Proof Since sq ^ soj Lemma | 6 ] gives sq ^ [sq], and we know that, by construction, sq = [sq]. 

Besides preserving alikeness, the construction also yield bi-simulating machines. 

Lemma 7 Let M he a FSM and let M he the FSM in Definition \11[ Then, M and M are bi-similar. 
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Proof Define the relation i? C S' x S' by letting (s, [r]) G i? iff s ~ r. Clearly, (sq, [sq]) & R- Now, let 
(s, [r]) G R with s —5> p for some p ^ S, x a O. Since s ^ r, Lemmami) gives r —)• g for some g G S 
with q ^ p. Then Fact Ogives [r] [g]. But {p, [g]) G R, and we conclude that R is a simulation relation. 

For the other direction, define the raletion L C S x S where ([r], s) G T iff r ~ s. Again ([sq], So) G L clearly 
holds. Let ([s],r) G L with [s] [g] for some [g] G S, a G O, a; G X. By FactO we get si —>■ gi for some 

si,gi G S with s ~ si and g ^ gi. Since ([r],s) G L, we have s ^ r, and so r ^ si. From si gi we 

conclude that r ^ g 2 , for some g 2 G S with g 2 gi, using Lemma [2ljl). Thus, g 2 ~ g, and so ([g],g 2 ) G L, 
and we conclude that L is also a simulation relation. □ 

The desired converse to Corollary [T] can now be established. 

Corollary 4 Let M be a p-reduced FSM and let T be a test suite for M. Assume that all p-reduced T-alike 
FSMs are isomorphic to M. Then T is perfect for M. 

Proof In view of Theorem[Tl it suffices to show that any FSM that is T-alike to M is also bi-similar to M. 
Let N be T-alike to M. Let N be as in Definition IIII By Corollary [2] is p-reduced, and by Corollary [3] we 
have N ^ N. Now, in view of Remark[^2) we conclude that N N- Since we already have M ^ N, using 
Lemma [Hand Remark |3l we conclude that M N. So, N is p-reduced and T-alike M. By the hypothesis 
we know that M and N are isomorphic. Hence, using Theorem [H we know that M and N are bi-similar. 
But N and N are also bi-similar, using Lemma 0 Finally, using Fact IH we conclude that M and N are 
bi-similar, as desied. □ 

We can now collect the results of this section in the following theorem. 

Theorem 3 Let M be a p-reduced FSM and let T be a test suite for M. Then T is perfect for M iff all 
p-reduced T-alike FSMs are isomorphic to M. 

Proof Use Corollaries [T] and S) □ 

4 Completeness and Perfectness 

In this section we investigate the relationship between completeness and perfectness. We show that a test 
suite T that is not n-complete for a FSM M can not also be perfect for M, for any n > I. In the other 
direction, we also show that there are test suites T which are perfect for M, but not n-complete for M, for 
n >2. 

We start by showing that perfectness only holds when n-completeness also holds. Let M be a FSM and 
let T be a test suite for M. We want to prove that if T is not n-complete for M, then T is not perfect for 
M, where n > I. This will show that perfectness is at least as strong a condition as is completeness. 

First, we need a measure on the length of blocking test cases in a test suite. Let a G T* be an input 
string for M. Define F{M,a) as: 

F{M, a) = max {|/3| : a = Pxj, with (3 G U{so), ftx ^ U{so), x G T}. 

That is, F{M, a) is the maximum length of a prefix of a which does not block in M. For a test suite TCI* 
we overload the notation and define F{M,T) = ^ F{M,a). 

aeT 

Fact 7 Given a FSM M and a test suite T for M, we have the upper bound F{M,T) < ^ |a|. 

aeT 


Proof Immediate. 
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Now, fix a FSM M, a test suite T, and assume that T is not n-complete for M, for some n > 1. Then, 
there is a FSM N such that M ^ N and M N. So, we have some a = X 1 X 2 ■ ■ ■ Xn+i, where n > 0 and 
Xi € X (1 < i < n + 1), and such that 


a^T and aGU{so)- 


Let 


xijai 2 : 2/02 Xn/a„ 

So Si —>■ S2 • ■ • Sn-1 Sn S„+i. 

We show how to construct a sequence of FSMs W that satisfy, for all z > 0: 


( 1 ) 

( 2 ) 


1 . Ni'is a tree rooted at go- 

2 . CT e C/i((?o)- 

3. for all a G Ui{qo) fl T we have: 

(a) a € C/(so). 

(b) If qo “4^ and sq then w = 77 . 

In order to ease the notation, we denote the states in each Ni as qo, qi, q 2 , ■■■, with qo the initial state. 
Moreover, by Ui{qo) we mean the set of all input strings a such that qo for some output string u). 

We start by defining No as the FSM containing the transitions: 


X\ ja\ 'X.'llcil Xnlo-n 21 ti . + i /& 

90 q2---Sn-l q-a ^ <ln+l. 


(3) 


where b ^ Un+i- It is clear that No is a tree rooted at qo, and that a G Uo{qo), and so properties (I) and 
(2) hold for Nq. Now, let a G Uo{qo) H T. Since a ^ T, we conclude that a is a prefix of xia :2 • ■ • x„, and so 
property (3) also holds for No- 

Now assume that W has been constructed satisfying properties (I)-(3), for some z > 0. If there is some 
input string a G U (sq) fl T such that a ^ Ui{qo) we show how to construct Since a ^ Ui{qo), we can 

write a = yiy 2 • • • VkxP, where fc > 0, z/j G I (I < j < k), x G I, and where we also have z/iz /2 ■ ■ - yk & Ui{qo), 
2 / 12/2 • • • VkX ^ Ui{qo). So, in Ni we have the transitions 


VI/bi yi/h^ 

To ri r2 • • • Tk-i 


Vklbk 

-t rk 


(4) 


with xq = qo and with no transition out of on input x. Since a G U{so), in M we get 

Vl/bi 1 / 2/62 Vk/bic xjc 

PO ^ Pi ^ P 2 ---Pk -1 -t Pk ^ Pk+ 1 , 


(5) 


for some c G X and with po = sq. We define W+i from Ni by adding to it a transition Vk —>■ r, and where r 
is a new state not present in Ni. 

Since W is a tree rooted at qo, then so is because r is a new state. Then property (I) holds for 

Ni+i. Also, since all transitions from Ni are present in W+i, then property ( 2 ), trivially, also holds for A^z+i. 
Now, let 7 G Ui+i{qo) n T. Since 7 G Ui+i{qo) we have two cases: 

• Case I: the new transition rk r does not occur in 7 . Then, clearly, 7 G Ui{qo), and so (3a) and 
(3b) hold because W satisfies property (3). 

• Case 2: the new transition r/c -4 r occurs in 7 . Since r is a new state, we can write 7 = 5x, where 
b G Ui{qo) and qo rj, ->■ r. Since Ni is a tree rooted at qo, there is only one path from qo to r^. 

Ni +1 Ni+i 

Hence, from Eq. dH) we get S = yiz /2 ■■■yk, and y = 61&2 ■■■bk- From Eq. © we get sq '^4 ''pk "4° pk+i, 
and property (3) holds for Ni+i. 
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We conclude that properties (l)-(3) hold for as desired. 

Because a = yiy 2 ■ ■ ■ UkxP, yiy 2 ''-ykX 0 Ui{qo) and the construction of gives yiy 2 ''-ykX G 

Ui+i{qo) we conclude that F{Ni,a) < F{Ni+i,a). Since we also have a € T, we then get F{Ni,T) < 
FiN,+i,T). 

The preceding discussion shows that we can construct the sequence of FSMs Nq, Ni, ... satisfying prop¬ 
erties (l)-(3), and with F{Ni,T) < F{Ni+i,T), as long as we have input strings G U{so) fl T such that 
cti ^ Ui{qo), i > 0. 

Fact 8 There is some £ > 0 such that there is no a G U{so) H T and such that a 0 Uelqo). 

Proof Fact [7] establishes an upper limit to the sequence F{Nq,T) < F{Ni,T) < • • •. □ 

Now we can take the test case a, that is not in T, and use the fact that the construction gives a € U(qe) 
to show that T is not, in fact, perfect for M. 

From Eqs. m and @ we can write sq where oj = 0102 • • • a„. From Eq. m and property (2), 

we get So Since a„+i ^ b we conclude that M Ni. If T was perfect for M we would have M /t Ni- 

Ni 

We now show that this leads to contradictions. There are two cases: 

• Case A: there is some input string a G U{so) fl Ue{qQ) fl T such that sq qo 
This contradicts property ( 3 b). 

• Case B: there is some input string a G (Cf(so) 0 Ue{qo)) CiT. li a € Ue{qo) Cl T 
contradict property (3a). If a G t7(so) fl T and a ^ Ui{qo), we contradict Fact [51 

We conclude that T is not perfect for M. 

Fact 9 Let M be a FSM, and let T be a test suite that is not n-complete for M, for some n > 1. Then, T 
is not perfect for M. 

Proof From the preceding discussion. □ 

Next we also show that when T is n-complete for M, n > 1, it may be the case that T is not perfect for 
M. Let the input and output alphabets be I = O = {0,1}, and let M be the specification with n states 

given by the transitions Si ^ s^+i, 0 < i < n. Let T = {0"',0"“^} be a test suite for M. We argue that 
T is n-complete for M. From Definitions [3] and HI if that were not the case, we would have a FSM N with 
17(so) C U{qo), and such that M ^ N and M N. Since C/(so) C U{qo) and U{so) = {0"“^}, we get 
U{so) n U{qo) r\T = {0"“^}. Hence M ssy N gives A(so,0"“^) = 0"“^ = /x(go,0"“^). Since we also have 
17(so)nC/(go)nI* = {0"“^}, Definition [3] and M N would require A(so, a) ^ h-iqo, ct) for some a G 
and we reached a contradiction. 

We now argue that T = {0", 0"“^} is not perfect for the same specification M. Let N be the FSM with 

the transitions qt ^ qi+i for 0 < i < n, and also qn-i ^ qn-i- It is clear that G ([/(sq) Ql7(qo)) . 

Hence, from DefinitionjSl we see that M N. Since T = {0",0”“^}, it is clear that {U{so)QU{qo))nT — 0. 
Moreover, U{sq) fl U{qo) fl T = {0"“^}, and so A(so,Q!) = y{qo,a) for all a G U{so) H U{qo) fl T. From 
Definition [5] we get M N. Hence, Definition [6] says that T is not perfect for M. 

Corollary 5 Let M be a FSM. Then the following holds: 

1. If T is a test suite which is perfect for M, then T is also n-complete for M, for all n > 1. 

2. For all n > 1 there are test suites which are n-complete but not perfect for M. 

Proof From the preceding discussion. □ 


a/ui2 1 , 

^ , and wi A W 2 . 

IS I 

and a ^ U{so), we 
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5 Test Suite Completeness and the Size of Implementations 

In this section we show that if one allows for too large implementations, then test completeness, in the 
classical sense, is lost. More specifically, if T is a test suite for a FSM M, then T is not n-complete for M, 
where n > k\S\ is the number of states in implementation machines, and fc is a constant that depends only 
on T. This means that T may not be able to detect all faults in implementations with n or more states. In 
the sequel, we use this result to also establish a bound on the size of implementation models when testing 
in the presence of blocking test cases, i.e., when testing for perfectness. 

First, we establish some notation. Let a = xqXi ■ ■ ■ Xk be a sequence of symbols over an alphabet. Then 
o'i,j (0 < I < j < fc + I) indicates the substring XiXi+i ■■■xj-i. Let a be another sequence of symbols 
over the same alphabet. We say that a is embedded in a if and only if there are sequences of symbols Pi 
(0 < f < fc + 1) such that a = PqXqPiXi ■ • • PkXkPk+i- Let T be a test suite for a FSM M and let a €T. We 
say that a is extensible in T if and only if cr = cricr 2 and there is some non-null 7 such that cri 7 cr 2 is in T. 
Otherwise, a is non-extensible in T. 

From this point on, we fix a reduced FSM M and a test suite T for M. Also, we fix ct = x^xi ■ ■ ■ Xk, k > 0, 
as a smallest non-extensible test case in T. Trivially, such a test case always exists. The following construc¬ 
tion, and the series of accompanying facts, will give us the desired result about the size of implementations 
when testing for completeness by. 


Remark 6 If T r\U{so) = 0 then any FSM is trivially T-equivalent to M. Moreover, if a = e, then T = {e} 
and, again, any FSM is trivially T-equivalent to M. Since M is redueed, one can easily eonstruct a one-state 
FSM that is not equivalent to M. Henee, in both cases, T would not be 1-eomplete for M. We, therefore, 
ean assume that sueh a non-null a GT Ci U{so). 

Since a G U(so), we get transitions TTi : Si Si+i in M if) <i < k). Those are the distinguished transitions 

of M. Moreover, since M is reduced, using Remark [5] we have Sk+i ^ s' in M, for some z G I, a G O and 
s' G S. We call this the marked transition of M. 

We now construct a FSM N using the same input and output alphabets, respectively T and O, of M. 
A simple example illustrating the construction is presented right after Theorem 2] Let Q = 5” x [0, fc + 1], 
that is, the states of N are pairs [q, i] where g is a state of M and 0 < i < fc -I- I. The initial state of N is 
go = [so) 0]- We complete the specification of N by listing its transitions: 

(a) If s r is not a distinguished transition of M, let [s, i] ^ [r, i] be a transition in N, for alH, 0 < f < k. 

(b) For all distinguished transitions Si —>■ Si+i of M, let [si, j] —>■ -I- 1] be a transition in N. We 

call these the distinguished transitions of N. 

(c) If s ^ r is not the marked transition of M, we let [s, k -\-1]^-^ [r, fc -I- 1] be a transition in N. 

(d) For the marked transition of M, Sk+i ^ s', we let [s^+i, fc -|- I] ^ [s', fc -|- 1], for some 6 ^ a, be a 

transition in N. 


This completes the specification of N. Easily, N has (|ct| + I)|S'| states. 

The next facts make explicit the behavior of the construction. 

Fact 10 Let tt : s -U p in M and take 0 < z < fc + 1. Then in N we must have [s,z] A [p,j\ for some 
j > i. Moreover, uj = uj' if the marked transition of M does not oecur in tt. 

Proof By induction on jaj = n > 0. When n = 0 the result follows immediately. 

For the induction step, let a = /3x, lv = pa, with x G T, a G O, and tt : s ^ r -U p. The induction 
hypothesis gives tti : [s, i] [r,j\ in N , with j > i. 
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If j = k + 1, then items (c) and (d) in the construction of N give [r,j] [p,j] in N. Then, clearly, 

[s,i] A- [Ptj\ in TV, where w' = p'a'. Moreover, if the marked transition of M does not occur in tt then 

the induction hypothesis gives p = p'. Also, since r -4 p is not the marked transition of M, item (c) of the 
construction of N yields a' = a. We conclude that oj = pa = p'a' = w', as desired. 

Now take j < k + \. Then items (a) and (b) of the construction give [r, j] [pA] in N where i = j or 

£ = j + 1. Hence, [s, i\ A- [p,j] with lu' = p'a' and, in any case, i > j > as desired. Again, if the marked 
transition of M does not occur in a then we get p = p' using the induction hypothesis. Clearly, from items 

(a) and (b) we have a' = a. This readily gives oj = pa = p'a' = uj', concluding the proof. □ 

The next result gives the converse. 

Fact 11 Let tt : [s,i] A [p,j] in N. Then we have: (i) j > i, (ii) aij is embedded in a, and (Hi) s ^ p 
in M. Moreover, lo = uj' if the marked transition of N does not occur in tt. 

Proof By induction on jal = n > 0. When n = 0 the result follows easily. 

For the induction step, let a = fix, uj = pa, with x G I, a € O, and tt' : [s,i] [r,i] A [p,j]. 

The induction hypothesis gives £ > z, embedded in /3, and s ^A r in M. Following the items in the 
construction of N we have four cases for the transition [r, (] A* [p, j]: 

(a) It was added because of item (a). Then, £ = j and r A p is in M. We get j = 1 > i and aij = ai^e is 

embedded in a, as desired. Composing we get s ^ A p in M, with j3x = a and p'a = uj'. If the marked 

transition of M does not occur in tt, then p = p' hy the induction hypothesis. So, uj = pa = p'a = uj', as 

we wanted. 

(b) It was added because of item (b). Then, x = xi, j = £+1, and r A p in M. Clearly, (i) and (iii) hold, 

with uj' = p'a. Also, Gip = = Oi^nxi. Since a = fdx = j3xi and Gi^i is embedded in j5, we conclude 

that (Tij- is embedded in a. If the marked transition of M does not occur in tt, then we proceed as in case 
(a), and obtain uj = pa = p'a = uj', as needed. 

(c) It was added because of item (c). Now we have £ = k + 1 = j and r A in M, showing that (i) and 

(iii) hold with s ^A p and uj' = p'a. We have that = Gij is already embedded in /3 and so its also 
embedded in a, given that a = jdx. The reasoning to obtain a; = w' is the same as in case (a). 

(d) It was added because of item (d). Proceed exactly as in case (c). Now, the marked transition of N does 
occur in tt and so the last statement of the Fact holds vacuously. This last case concludes the proof. □ 

The last two results already establish that the same sequences of input symbols will run in both machines. 


Fact 12 U{so) = U{qo). 

Proof Recall that go = [sojO]. Let sq A in M. Using FactfTUlwe get [so, 0 ] A in N. Hence, U{so) C [/(go)- 
In a similar way we can get [/(go) C U{so) using FactfTTl and the result follows. □ 

We are now in a position to show that M and N are T-equivalent. 

Fact 13 M N. 
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Proof We go by contradiction. Assume we have ax G T D U{so) n U{qQ), x such that Sq s ^ r in 

Oi j UJ 3^ I \) GL j GL j 

M and [so,0] — >■ [q,i] — >■ [p,j] in N, with a ^ b. Fact ITT] gives So g in M. But we already have sq — t s 
in M, and so we conclude that s = q. Using Fact [TT] again, from s%- r in M and [s,i] ^ [p, j] in N we get 

3^ j 3^ j Oj 

p = r. We can now write tt : [s,z] [r, j] in N and s ^ r in M with a ^b. From the construction of N 

we conclude that tt is the marked transition of N. Hence, i = j = k + 1. We now have [sq, 0] [s, A: + 1] 

in N. From Fact m a = (To,fc+i is embedded in a and so a is embedded in ax. Since ax € T, we conclude 
that cr is extensible in T. But this contradicts the choice of tr, completing the proof. □ 

In the opposite direction, the next result shows that M and N are not equivalent. 

Fact 14 M ^ N. 

Proof Since a € U{so), Fact [T^ gives a € U{qo). By the choice of cr, in M we have Sq Sk+i- Further, by 

the choice of z and a, we have s^+i ^ s' in M. Hence, sq in M. Item (b) of the construction of N 

gives [si, i] [si+i, * + I], 0 < i < A. Then, [sq, 0] [s^+i, fc + I] in N. By item (d) of the construction 

of N we get [s^+i, fc + I] [s', fc + 1] in N. Composing, we obtain [sq, 0] [g'^ A; + I] in N. This shows 

that M ^ N, because a ^ b. □ 

Collecting, we can show that a test suite T will not be n-complete for a FSM M when n is larger than a 
certain bound, which depends only on M and T. 

Theorem 4 Let M be a FSM and let T be a test suite for M. Let a be a shortest test case in T that is 
non-extensible in T. Then T is not ((|cr| + l)|S'|)-co? 7 ipteAe for M. 

Proof The construction of N yields a machine that is T-equivalent to M, using Fact [131 We also know that 
M and N are not equivalent, by Fact [TH Also, using Fact [T^ we know that U{sq) C [/(go)- Since N has 
n = (|cr| + 1) X [S'l states. Definition |T| says that T is not n-complete for M. □ 

Next, we give a simple example to illustrate the construction of machine N. Let M = {S, sq,!, O, D, S, A) 
be a specification FSM as depicted in FigurejH The set of states is S' = {sq, si}, T = O = {0,1}, and AA, 6 , A 
are given as depicted in the figure. Note that M is a partial FSM since (si, 1) ^ D. Also let T = {0000,100} 
be a test suite for M. We notice that T is 2-complete for M, i.e., for implementation FSMs with at most as 
many states as M. This can be checked by using the algorithm described in [BM14bl IBM13] . 

Now take a = 100 as the shortest test case in T that is non-extensible in T. We apply items (a) to 
(d) of the construction of N, thus obtaining a machine with (|(t| -|- 1)|S| = (3 -I- 1)2 = 8 states. From item 

(a) we create transitions [sq,*] [sou]; for all i, 0 < i < 2. We also obtain the distinguished transitions 
[so,0] [si,l], [si,l] [si)2], [si,2] [si,3] [so,l] [si,2], [so,2] [si,3] and [si,0] [siU] from 

item (b). From item (c) we get the transitions [so,3] %• [soj3], [so:3] % [s 0 ) 3 ] and [s 0 ) 3 ] ^ [si,3]. Finally 

we complete machine N with the marked transition [ 53 , 3] [ 53 , 3] as required by item (d). Machine N is 

depicted in Figure |3l It is a simple matter to see that states [sq, 1], [so: 2], [sq, 3] and [si, 0] are not reachable 
in N. Then we can remove them in order to obtain a reduced FSM as depicted in Figure H] Note that we 
have renamed states as qo = [so, 0], qi = [si, 1], 52 = [si, 2], and q^ = [si, 3]. 

Now we can easily check that M rst N because A(so, 0000) = 1111 = r(qo, 0000) and A(so, 100) = 100 = 
T(go, 100). But M ^ N since we have A(so, 1000) = 1000 ^ 1001 = T{qo, 1000). It is also easy to verify that 
U{so) U U{qo). We conclude that T is not 4-complete for M, and so it is also not 8 -complete for M, where 
8 is the bound specified by Theorem [H 
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Figure 3: A candidate implementation N. 



Figure 4: A reduced candidate implementation N. 


6 m-Perfectness 

Combining Theorem 0] and Corollaryj^l), we see that no test suite T can be perfect for a given specification 
M if we allow the number of states in implementations to be put under test to grow beyond a bound fc|S'|, 
where is the number of states in M and fc is a constant that depends on T alone. This leads us to the 
notion of m-perfectness. 

Definition 13 Let M be a FSM and T be a test suite for M. Then T is m-perfect for M iff for any FSM 
N with at most m states, if M N then M 'fx N. 

That is, m-perfectness guarantees that any difference in behavior between the specification M and a imple¬ 
mentation N will be detected when we run the tests in T, even in the presence of blocking test cases, given 
that implementations are restricted to have at most m states. In other words, if T is a m-perfect test suite 
for a specification M, then for any implementation under test N, if M and N are unlike, then they are also 
T-unlike, provided that N has at most m states. 

We proceed to obtain necessary and sufficient conditions for m-perfectness, by showing that a result 
analogous to Theorem [T] The following result will be useful when we consider certain bi-similarities. 

Lemma 8 Let M and N be FSMs. Let n > 1, Si € S, pi G Q (I < i < n) and Xi Gl, at G O, bi G O' 

(1 < i < n) be such that Si —>■ and pi Pi+i (1 < i < n). Assume further that si ~ pi. Then 

Si Pi (i < i < n) and 0102 •• • a„_i = &162 • • • &n-i • 


Proof Let a = X 1 X 2 ■ ■ ■ Xn-i, wi = 0102 ■ • • On-i and a ;2 = & 1&2 ■ • ■ &n-i- We clearly have si Sn and 
Pi Pn- Definition [ 5 ] immediately gives wi =012, because si ~ pi and a G U(si) H U{qi). 
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To see that Si ^ pi {1 < i < n) we go by induction on n. The basis follows from the hypothesis, and 
we proceed with the induction step. Let 1 < A: < n and assume Sk ~ Pk- Let a = Xi---Xk- Clearly 
(5(si,a) = Sfc+i, p{pi,a) = Pk+i and so a € U{si) fl U{pi). For te sake of contradiction, assume that 
Sk+i Pk+i- By Definition [S] we have two cases. 

Case 1: U{sk+i)QU{pk+i) 

Let /3 e U{sk+i) and /3 ^ U{pk+i)- This gives a/3 € U{si) and a/3 ^ U{pi). Hence U{si) © U{pi) ^ 0, 
contradicting si ^ p\. The situation when /3 ^ U{sk+i) and /3 € U{pk+i) is entirely analogous. 

Case 2 : /3 e U{sk+i) r\U{pk+i) and A(sfc+i,/3) # r(pfe+i,/3), for some /3 e J*. 

This gives a/3 G U{si) Cl U{pi). Moreover, 

A(si,a/3) = A(si,a)A(d(si,a),,5)) = A(si,a)A(sfe+i,,5), and 
T{pi,a(3) = r(pi,a)r(^(pi,a),/3)) = t(pi, a)r(pfe+i,/3). 

Because |A(si,a)| = |T(pi,a)| and A(sfc+i,,5) 7 ^ r(pfc+i,/3), we get A(si,a/3) 7 ^ r(pi,a/3). Since a/3 e 
U{si) riU{pi), this contradicts si ~pi. 

The proof is complete. □ 

The next result guarantees the existence of bi-simulations in the presence of blocking test cases. 

Lemma 9 Let T be a m-perfect test suite for a FSM M. Let N he a FSM with at most m states such that 
M '^T N. Then M and N are bi-similar. 

Proof Define a relation Ri C S x Q hy letting {s,q) G Ri if and only if d(so,a) = s and p{qo,a) = q for 
some a G X*, s G S and q G Q. Since (5(so! £) = so and p{qo, e) = <70 we get (so; 9o) € ^i- 

Now assume (s, q) G Ri and let s ^ r for some r G S, x G I and a G O. Since (s, q) G Ri, the definition 
of i?i gives some a G I* such that (5(so, a) = s and p(cio, a) = q. Composing, we get (5(so, ocx) = d{s,x) = r 
and so ax G U{so). Since T is m-perfect for M and M 3V, Definition IT^ gives M ^ N, that is sq ~ go- 
Further, Definition [5] and Remarkimply 17(so) = U{qo), and so ax G (7(qo)- Then p{q,x) = p, for some 
p G Q. Since Sq ~ go; <J(soiQ^) = s and /r(go,a) = g. Lemma |5] gives s ~ g. But x G t/(s) fl 17(g), and 

so we must have a = X(s,x) = T{q,x). Thus, we have found p G Q with q -U p. Since 8 [senax^ = r and 
fi{qo,ax) = p, we also have {r,p) G Ri- This shows that Ri is a simulation relation. 

A similar argument will show that R 2 F Q x S, where R 2 = Rf^, is also a simulation relation. Thus M 
and N are bi-similar, as desired. □ 

We now show the converse, that is, if M is bi-similar to any FSM N with at most m states that is T-alike 
to it, then T is a m-perfect test suite for M. 

Lemma 10 Let M be a FSM, T a test suite for M, and m > 1. Assume that any FSM that is T-alike to 
M with at most m states is bi-similar to it. Then T is m-perfect for M. 

Proof We proceed by contradiction. Assume that T is not m-perfect for M. Then, by Definition [131 there 
exists a FSM N with at most m states such that M N and M N. Hence, since M N, by TheoremjT] 
we know that N is bi-similar to M, and so we have simulation relations Ri Q S x Q and R 2 Q Q x S. 

Since M / N, by Definition [5] we have two cases: 

Case 1: a e U{so) 0 U{qo), for some a G T*. 

We may assume that |a| is minimum. If a G U{qo) and a ^ Lf{so), then we may write a = fix, where 
/3 € X*, X G I are such that /3 G U{qo) Cl C/(so)- Thus, S{so,fi) = s, fJ.{qo,fi) = q and p{q,x) = p, 
for some s G S and some q,p G Q. Since (go,so) G R 2 , we can use Lemma [ 8 ] and write (g, s) G i? 2 - 
Because R 2 is a simulation and p.{q,x) = p we get some r G S such that d{s,x) = r. But this gives 
(5(so,a) = S(so,fSx) = S(s,x) = r, that is a G U{so), a contradiction. When a ^ U{qo) and a G U{so), 
the argument is analogous. 
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Case 2: There is some a € U{so) fl U{qo) with A(so,a) ^ r(go,a). 

Again, assume that |a| is minimum. Then, there are s^S and q € Q such that a = j3x 

and d{so,l3) = s, /r(go,/3) = 9- Further, we get some r G S, p G Q such that (5(s,a:) = r, p{q,x) = p, 

and a = X{s,x) ^ T{q,x) = b. Using the Lemma HI we may write (5,9) G Ri- Because we have s r 
in M and Ri is a simulation, we know that there is some t G Q such that q t in N, with (r, t) G Ri- 

X j b 

But we already had q p in N. Hence, since N is deterministic, we conclude that a = b, which is a 
contradiction. 

The proof is now complete. □ 

Combining the previous results we obtain necessary and sufficient conditions for m-perfectness. 

Theorem 5 Let M be a FSM, T be a test suite for M, and m > 1. Then T is m-perfect for M iff any 
T-alike FSM with at most m states is bi-similar to M. 

Proof Assume that T is m-perfect for M. Lemma O guarantees that N and M are bi-similar when N is 
T-alike to M. Now assume that any T-alike FSM with at most m states is bi-similar to M. In this case. 
Lemma fTOl guarantees that T is m-perfect for M. □ 

7 Conclusions 

In this work we have studied the notion of test suite perfectness, a notion similar to the classical one of test 
suite completeness, but now we may have the presence of so called blocking test cases, that is, test cases 
that may not run to completion either in the specification or in implementation models. An accompanying 
notion of p-reduction was also introduced, similar to the classical notion of reduction in FSMs. 

We showed that any FSM can be p-reduced while maintaining the perfectness property, when it was 
already present in the original FSM. Using this result, we then proved that when the specification model and 
implementations to be put under test are both p-reduced, then perfectness can be characterized in terms of 
an isomorphism between both models. 

We then established the relationship between perfectness and the classical notion of completeness. We 
showed that perfectness is a strictly stronger relation, for specifications models of any sizes. We then showed 
that when testing for perfectness one has to impose a limit on the number of states of the implementation 
models that are put under test. This result was a consequence of a similar bound of the form kn that 
we showed must be imposed on the size of implementations when also testing for the classical notion of 
n-completeness. Here, fc is a constant that depends only on the test suite and n is the number os states in 
the specification model. 

We then characterized the m-perfectness property by establishing a necessary and sufficient condition on 
the implementation models that are put under test, given a test suite and a specification model. 

For future studies, we mention developing and testing a practical algorithm for testing m-perfectness. 
Further, it may be the case that one can obtain tighter bounds on the size of implementation models when 
testing for either m-perfectness or for n-perfectness. 
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